Moving image data management apparatus and method

ABSTRACT

A moving image data management apparatus which manages one or plural representative images in correspondence with a moving image recorded on a storage medium. The apparatus generates a thumbnail based on a frame image selected in a desired selection base in an arbitrary section designated in the moving image, in correspondence with the section, and records the thumbnail. The thumbnail, corresponding to the moving image, is stored in correspondence with section information indicating the designated section, position information indicating the position of the selected frame image, and Type ID indicating the selection base. Among the thumbnails registered in this manner, a thumbnail is selected by referring to the Type IDs in correspondence with a processing purpose and is presented to a user. As the selection base which is information on generation of thumbnail is held as an attribute of the thumbnail, and a thumbnail appropriate to processing can be presented, operability in processing using thumbnail can be improved.

This application is a continuation of Application No. 11/126,363 filedMay 11, 2005, now U.S. Pat. No. 7,155,064 which is a continuation ofApplication No. 10/845,198 filed May 14, 2004, now U.S. Pat No.6,965,701 which is a continuation of International Application No.PCT/JP02/12921 filed Dec. 10, 2002.

TECHNICAL FIELD

The present invention relates to moving image data management apparatusand method for management of moving image data recorded on a storagemedium by one or plural representative images.

BACKGROUND ART

Conventionally, a method of managing moving images recorded on a storagemedium by using representative images has been proposed. Japanese PatentNo. 03081304 discloses using representative images in generalmoving/still image reproduction apparatus, and especially disclosesusing representative images for simplified display upon selection ofstill or moving image. Further, Japanese Published Unexamined PatentApplication Nos. Hei 11-176137 and 2000-322875 disclose assigning arepresentative image to a moving image section, and selecting a sectionas a target of reproduction designation by using the representativeimage.

As such representative image is generally used, variety of usage ofrepresentative image increases. For example, image management using therepresentative images can be applied to an editing apparatus as well asthe above-described reproduction apparatus.

However, even regarding the same section in a moving image, arepresentative image appropriate to a user interface upon reproductionand a representative image appropriate to a user interface upon editingare different from each other. Accordingly, if the same representativeimage is to be used in various apparatuses including an editingapparatus, the following problem occurs.

For example, in FIG. 1, reference numeral 11 denotes a moving image. Inthis moving image 11, in a section (a), the motion of a batter who holdsa bat at the ready and hits a ball is recorded. Numerals 12 and 13denote representative images regarding the same section (a). Therepresentative image 12, generated from a head frame of the section (a),is an image of the batter at the ready. The representative image 13,generated from a middle frame of the section, which is an image of thebatter when he hits the ball. The representative image 13 can be a mostappropriate representative image to indicate the content of the section(a).

Generally, one representative image is assigned to one section and isregistered. Accordingly, one of the representative images 12 and 13 isactually registered as a representative image of the section (a). Theregistered representative image can be used as an index image for e.g. areproduction apparatus. However, if the representative image 13 isregistered, upon selection of representative image in the reproductionapparatus, reproduction starts from a status prior to the hit (thereproduction starts from the head of the section (a)), thus a user feelsunnatural. On the other hand, if the representative image 12 isregistered, when the image is used in an editing apparatus, it isinconvenient that the content of the section (a) cannot be preciselygrasped by the image 12. That is, in the example of FIG. 1, it cannot bedetermined from the representative image 12 whether the section (a)indicates a scene where the batter hits the ball or a scene where thebatter swings at the ball and miss. In this manner, an appropriaterepresentative image in one section of moving image depends on thepurpose of apparatus to be used.

Conventionally, assignment and management of representative images arenecessary in accordance with purpose and usage of moving image data suchas reproduction and editing. Accordingly, information on therepresentative images are redundantly held by purpose such as editing orreproduction, by moving image or still image, or by apparatus or usage.Further, program code for processing to utilize the information on therepresentative images must be respectively held for the representativeimages. That is, when the representative images are used in variousmoving image data processing, the information on the representativeimages are respectively managed by apparatus or program. Accordingly, itis wasteful to have plural apparatuses and programs for management, andit is further wasteful to independently store the representative imagedata. To maintain operability in moving image data processing upon useof representative image, management of representative images becomescomplicated.

DISCLOSURE OF INVENTION

The present invention has been made in consideration of the aboveproblem, and has its object to improve operability of various movingimage data processing using representative images and to simplifymanagement of the representative images by adding information regardingselection of frame as an original of the representative image as anattribute of the representative image to the representative image.

According to the present invention, the foregoing object is attained byproviding an apparatus for managing one or plural representative imagesin correspondence with a moving image, comprising; means for storing arepresentative image, generated based on a frame image selected from themoving image, in correspondence with the moving image; and means forstoring section information indicating a section in the moving image andattribute information indicating a selection base of the frame image, incorrespondence with the representative image.

According to another aspect of the present invention, the foregoingobject is attained by providing a method for managing one or pluralrepresentative images in correspondence with a moving image, comprising;a step of storing a representative image, generated based on a frameimage selected from the moving image, in correspondence with the movingimage; and a step of storing section information indicating a section inthe moving image and attribute information indicating a selection baseof the frame image, in correspondence with the representative image

Other features and advantages of the present invention will be apparentfrom the following description taken in conjunction with theaccompanying drawings, in which like reference characters designate thesame name or similar parts throughout the figures thereof.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention.

FIG. 1 is an explanatory view of problem in utilization ofrepresentative image;

FIG. 2 is a block diagram showing a control construction of a movingimage processing apparatus according to an embodiment;

FIG. 3 is a block diagram showing a functional construction of themoving image processing apparatus according to the embodiment;

FIG. 4 is a schematic diagram showing an example of thumbnail storagestatus in a moving image file according to the embodiment;

FIG. 5 is a table showing an example of data structure for one index inan index area 406 in a footer 403 in FIG. 4;

FIGS. 6A and 6B are tables showing examples of data structure ofthumbnail 405 in a header 401 and one thumbnail in a thumbnail area 407in the footer 403 in FIG. 4;

FIG. 7 is a flowchart showing an example of thumbnail registrationoperation according to the embodiment;

FIGS. 8A and 8B are examples of user interface upon execution ofbrowser;

FIGS. 9A and 9B are flowcharts showing processing upon browser operationin the moving image processing apparatus;

FIG. 10 is an example of the user interface upon moving image editing byan editing unit 305;

FIG. 11 is a flowchart showing a thumbnail display operation upon themoving image editing by the editing unit 305; and

FIG. 12 is an example of division of frame into blocks.

BEST MODE FOR CARRYING OUT THE INVENTION

A preferred embodiment of the present invention will now be described indetail in accordance with the accompanying drawings.

In the present embodiment, in a method for utilizing representativeimages, to manage/utilize representative images and information relatedto the representative images in a simple manner, each representativeimage has information upon generation of the image. By using thisinformation, representative images appropriate to respective processingsare accessible for plural apparatuses and control methods, and arepresentative image to be utilized is selected in correspondence withpurpose or the like. Note that the information upon generation ofrepresentative image means data indicating an image in a correspondingmoving image section as an original of the representative image or thetype of image (a dynamic motion or the like) in the corresponding movingimage section. The information upon generation of representative imageincludes information regarding frame selection (selection base) as anoriginal of the representative image.

According to the present embodiment, as information upon generation ofrepresentative image is provided, plural representative images can berecorded and held in a centralized manner, thus waste is avoided (Forexample, plural representative images corresponding to plural types ofpurposes can be managed as one file). Further, as an appropriaterepresentative image to realize a particular function in each apparatusor program can be selected by referring to selection bases ofrepresentative images, the operability of the function can be improved.Further, upon storage of information, the arrangement of the informationor the like may be determined in correspondence with the purpose ormethod of the information regarding the representative image. That is,the locations of significant thumbnail (header), an index area and athumbnail area (footer) on a storage medium can be arranged toeasily-accessible positions in correspondence with significance.

Hereinbelow, a moving image processing apparatus according to thepresent embodiment will be described in detail.

FIG. 2 is a block diagram showing a control construction of the movingimage processing apparatus according to the present embodiment. In FIG.2, numeral 201 denotes a CPU which performs various controls in themoving image processing apparatus. Numeral 202 denotes a ROM in which aboot program executed upon boot-up of the apparatus and various data arestored. Numeral 203 denotes a RAM in which a control program forprocessing by the CPU 201 is stored, and which provides a work area forexecution of various controls by the CPU 201. Numeral 204 denotes akeyboard, and 205, a mouse. The keyboard and the mouse provide anenvironment for user's various input operations.

Numeral 206 denotes an external storage device comprising a hard disk, afloppy disk, an optical disk, a magneto-optic disk, a magneto-optictape, magnetic tape, a non-volatile semiconductor memory (e.g. a flashmemory) or the like. Numeral 207 denotes a display unit comprising adisplay of CRT, LCD or the like which displays results of variousprocessings and the like for a user. Numeral 208 denotes a networkinterface which enables communication with respective devices on anetwork. Numeral 209 denotes a video interface which reads a movingimage from a video camera 210 and a VTR 212. Numeral 211 denotes a businterconnecting the above respective elements.

Note that the video camera 210, the VTR 212 and the external storagedevice 206 may be substituted by devices arranged on the network.

FIG. 3 is a block diagram showing a functional construction of themoving image processing apparatus according to the present embodiment.Numeral 300 denotes a user operation unit which obtains the user'sdesignation or input via the keyboard 204, the mouse 205 and the like.Numeral 301 denotes a moving image input unit which reads a moving imagefrom the video camera 210, the VTR 212 or the like via the video I/F209, and inputs the image into the moving image processing apparatus.Numeral 302 denotes a moving image file storage unit comprising theexternal storage device 206, which holds moving images inputted by themoving image input unit 301 in file format.

Numeral 303 denotes a thumbnail generation unit which presents a movingimage stored in the moving image file storage unit 302 on the displayunit 207, such that the user designates a desired section and a frameimage in the section. The thumbnail generation unit 303 generates areduced image (hereinbelow, thumbnail image) based on the designatedframe image, and stores it, as a moving image file, with information onthe designated section or the like, into the moving image file storageunit 302. That is, by the thumbnail generation unit 303, the user canrefer to the moving image presented on the display unit 207, anddesignate a section in the moving image and a frame representing thesection via the user operation unit 300. Then the thumbnail image basedon the designated frame image is inserted into the moving image file,with the information on the designated section or the like, and the fileis stored into the moving image file storage unit 302. Note that theentire moving image may be designated as a section.

Numeral 304 denotes a browse unit which reads a thumbnail from themoving image file storage unit 302 and displays it via the display unit207, and reproduces a moving image corresponding to the thumbnailselected by the user. Upon reproduction of moving image in this case,only a section represented by the selected thumbnail is reproduced, orthe section represented by the selected thumbnail and the subsequentmoving image are reproduced. Numeral 305 denotes an editing unit whichperforms editing on a moving image held in the moving image file storageunit 302. Further, the progress and result of the editing are displayedvia the display unit 207.

Note that the functions of the above-described respective elements arerealized by execution of the control program stored in the ROM 202 bythe CPU 201. Otherwise, it may be arranged such that the control programis appropriately loaded from the external storage device 206 onto theRAM 203 and executed by the CPU 201.

<1. Thumbnail Storage Method>

FIG. 4 is a schematic diagram showing an example of thumbnail storagestatus in a moving image file according to the embodiment.

Numeral 400 denotes a moving image file stored on a storage medium by amoving image file storage unit 302. Note that the unit of storage is notlimited to 1 moving image file, but the unit may be a shot fromdepression of recording button of the image sensing device to stoppageof image sensing, or may be 1 TV broadcast program. The moving imagefile 400 comprises a header 401 in which management information or thelike is recorded, a moving image 402 in which moving image data itselfis recorded, and a footer 403 in which mainly a thumbnail is recorded.

In the header 401, moving image file management information 404 on thedata size, the format, the date of formation and the like of the movingimage file, and 1 image of thumbnail 405 are recorded. The footer 403has a thumbnail area 407 in which plural thumbnails can be recorded andan index area 406 in which index information on the respectivethumbnails are recorded.

FIG. 5 is a table showing an example of data structure for 1 index inthe index area 406 in the footer 403 in FIG. 4.

“Start Point” and “End Point” are start point and end point of a sectionin a moving image represented by the thumbnail. The form of the startpoint and the end point is not particularly limited as long as thepoints specify the section in the moving image, however, it ispreferable that the points are described in time code or frame ID. Notethat in a case where the entire moving image is designated as a section,the Start Point indicates a head frame of the moving image, and the EndPoint, an end frame of the moving image.

In “Thumbnail Type ID”, an ID is determined by image selection base upongeneration of thumbnail. In the present embodiment, Thumbnail Type IDsare determined for purposes of thumbnail as follows.

Thumbnail Type ID=1: head frame in a section

Thumbnail Type ID=2: middle frame in the section

Thumbnail Type ID=3: end frame in the section

Thumbnail Type ID=4: dynamic motion frame in the section

Thumbnail Type ID=5: gentle motion frame in the section

Thumbnail Type ID=6: object-appeared frame in the section

Thumbnail Type ID=7: object-disappeared frame in the section

Thumbnail Type ID=8: high image-quality frame in the section

Thumbnail Type ID=9: image not included in the section

In this example, IDs are defined based on types of frames in thesection. These IDs indicate selection bases of frames as originals ofthumbnails.

Further, in “Thumbnail Offset”, in FIG. 5, an offset to the thumbnailindicated by the index (the position of thumbnail corresponding to theindex in the thumbnail area) is described.

FIGS. 6A and 6B are tables showing examples of data structure ofthumbnail 405 in the header 401 and 1 thumbnail in the thumbnail area407 in the footer 403 in FIG. 4. In FIG. 6A, in “Thumbnail Header”,management information of the thumbnail image or the like is described.Further, in “Thumbnail Image”, the thumbnail image data itself isdescribed. Further, FIG. 6B shows an example of data structure of the“Thumbnail Header” shown in FIG. 6A. In “Thumbnail Image Offset”, anoffset to the storage position of the thumbnail image (Thumbnail Image)is described. In “Thumbnail Image Size”, the size of the thumbnail imageis described. In “Thumbnail Key Relative Point”, the frame position inthe moving image as an original of the thumbnail is described as arelative position from the “Start Point”.

Note that in case of the image not included in section, Thumbnail TypeID=9, as no image in the section is used, particular value such as amaximum value may be described.

<Thumbnail Registration Processing>

FIG. 7 is a flowchart showing an example of thumbnail registrationoperation according to the embodiment.

First, at step S100, the moving image input unit 301 inputs a movingimage, and records it in the format of the moving image file as shown inFIG. 4 into the moving image file storage unit 302. At this time, anarea for storing a thumbnail in the header 401 and an area for thefooter 403 are ensured as reserved areas. Next, at step S101, it isdetermined whether or not thumbnail designation has been made by theuser. The thumbnail designation means designation of arbitrary frame ina moving image as an original of the thumbnail, a section in the movingimage represented by the thumbnail, a selection base for selection ofmoving image frame as an original of the thumbnail, designation as towhether the thumbnail is significant or not, and the like. The selectionbase corresponds to the above-described Thumbnail Type ID in FIG. 5.Further, a significant thumbnail is a thumbnail which represents themoving image file and is registered as the thumbnail 405.

The thumbnail designation is made by the user by using functions of theapparatus. For example, in case where the user manually makes setups,(1) the user sets the apparatus to a predetermined mode for thumbnaildesignation, (2) then designates a desired section in the moving image,(3) then performs reproduction of the section, (4) and when an imagefrom which the user wants to generate a thumbnail is displayed, the userinputs a predetermined instruction into the apparatus. In this case, theabove Start Point and End Point are determined by the designated sectionin the moving image. Further, designation of desired section and imagefrom which a thumbnail is to be generated are made by using the keyboardand/or mouse. Further, by inputting a selection base to be used at thattime, the Thumbnail Type ID is determined.

Otherwise, it may be arranged such that a thumbnail is generated byautomatically selecting an image from the section designated by the userbased on the selection base designated by the user such as an image withdynamic change or an image with small change.

Further, it may be arranged such that a frame at the head/middle/end ofthe section is designated as a selection base, and if the “head” isdesignated, a head frame of the section is used, or if the “end” isdesignated, an end frame of the section is used, or if the “middle” isdesignated, a frame designated by the above procedure (1) to (4) isused, as a base image of the thumbnail.

As described above, the designation of moving image frame as an originalof thumbnail can be made by various methods such as the user'sdesignation and automatic designation by the apparatus. The presentinvention is applicable to any of the methods. However, in the flowchartof FIG. 7, a procedure using head, middle or end frame as a selectionbase is employed.

Next, at step S102, it is determined whether or not the designation thatthe thumbnail is significant has been made. As described above, in thepresent embodiment, the significant thumbnail means a thumbnail whichrepresents the entire moving image file. If the designation ofsignificant thumbnail has been made by the user, process proceeds tostep S103, at which the frame position designated for the generation ofthumbnail is corrected to a relative position from the head of themoving image file, to obtain the “Thumbnail Key Relative Point”. Next,at step S104, a thumbnail 405, having the “Thumbnail Key Relative Point”calculated at step S103, the size of the thumbnail image (ThumbnailImage Size), and information on relative distance to the storageposition of the thumbnail image (Thumbnail Image Offset) as a header, isrecorded into the header 401. At this time, the header 401 may be storedin a position for comparatively quick and easy access on the storagemedium, e.g., in use of optical disk, in an inner peripheral portion,thereby high-speed index display for moving image file selection can berealized. Then, the process returns to the determination at step S101.Note that as an important thumbnail is needed upon selection of movingimage file in a later operation, it may be arranged such that thepresent process does not end before the registration of significantthumbnail is completed (the process does not branch to YES at stepS115). In a case where no significant thumbnail is stored, it isapparent for persons skilled in the art that any one of thumbnailsstored in the footer, e.g., an initially-stored thumbnail (the head ofthe footer) may be stored. In such case, it is not necessary to checkthe completion of registration of significant thumbnail upon completionof the above processing.

On the other hand, if it is determined at step S102 that the designationof significant thumbnail has not been made, the process proceeds to stepS105. At step S105, a new index is ensured in an unused area of theindex area 406 in the footer 403 in the moving image file 400. Next, atstep S106, a new thumbnail is ensured in an unused area of the thumbnailarea 407. Next, at step S107, an offset value to the thumbnail ensuredat step S106 is calculated.

Next, at step S108, the type of designated thumbnail is determined basedon the selection base (in this embodiment, whether the position ofthumbnail base image is the head, middle or end of the sectionrepresented by the thumbnail) designated by the user.

The process branches to steps S109 to S111 in accordance with the typedetermined at step S108, and a corresponding Thumbnail Type ID is set.The setting of Thumbnail Type ID can be made by holding heabove-described Thumbnail Type IDs in a table and referring to thetable. In the present embodiment, if the selection base is the headframe in the section, setting is made as Thumbnail Type ID=1 (S109); ifthe selection base is the middle frame in the section, setting is madeas Thumbnail Type ID=2 (S110); and if the selection base is the endframe in the section, setting is made as Thumbnail Type ID=3 (S111). Inthis embodiment, the number of Types is three, however, the invention isnot limited to this number.

Next, at step S112, an index having the offset (Thumbnail Offset)calculated at step S107, the Thumbnail Type ID set at steps S108 toS111, and the Start Point and the End Point indicating the sectiondesignated by the user, is generated, and stored into the footer 403. Atthis time, the index area 406 of the footer 403 may be stored in aposition for comparatively quick and easy access on the storage medium,e.g., in use of optical disk, in an inner peripheral portion, therebyhigh-speed selection of thumbnail in footer can be realized. Further, itis preferable that the index area 406 is recorded in a continuous areaon the storage medium.

Next, at step S113, the position of the frame adopted as the thumbnailis corrected to a relative position from the header of the section, andthe Thumbnail Key Relative Point is obtained. At step S114, a thumbnailgenerated by reducing the designated frame is stored, with the ThumbnailKey Relative Point calculated at step S113 and the image size of thethumbnail (Thumbnail Image Size) as a header, into the footer 403, andinformation on a relative distance to the storage position of thethumbnail image is stored in the Thumbnail Image Offset. At this time,the thumbnail area 407 of the footer 403 may be stored in a position forcomparatively quick and easy access on the storage medium, e.g., in useof optical disk, in an inner peripheral portion, thereby high-speeddisplay of thumbnail in footer can be realized. Next, the processreturns to step S101, and if the designation of thumbnail is made again,the processing at steps S102 to S114 is repeated.

Further, if it is determined at step S101 that the designation ofthumbnail has not been made, the process proceeds to step S115, at whichif it is determined that termination of registration processing has beeninstructed, the process ends. If it is determined that the terminationof registration processing has not been instructed, the process returnsto step S101, to repeat the above processing.

By the above processing, any one of significant thumbnail, index andthumbnail image can be stored in a position for quick and easy access onthe storage medium. Further, as thumbnails for different purposes, orthumbnails for different device features though the purpose is same canbe discriminated by Thumbnail Type IDs, these thumbnails can be storedin the same file, thus management can be simplified.

<Thumbnail Utilization Processing>

Processing upon utilization of thumbnail registered by the abovethumbnail registration processing will be described.

[Utilization by Browser]

First, utilization of thumbnail in a browser will be described. FIGS. 8Aand 8B are examples of user interface of a browser. FIG. 8A shows adisplay status when an image moving file to be browsed has been selectedby a predetermined selection operation. Accordingly, the respectivethumbnails in FIG. 8A are registered in the footer of the selected file.The predetermined selection operation means, e.g., displaying a list ofthumbnails registered in the headers of the respective moving imagefiles as significant thumbnails as shown in FIG. 8A, and selecting amoving image file by designating a desired thumbnail. In FIG. 8A,numeral 801 denotes an entire display area of the browser. Numeral 802denotes an area where arrayed thumbnails are displayed. The thumbnailsare displayed from the top line, from the left to the right. Numeral 803denotes a scroll bar. In a case where the thumbnails cannot be displayedwithin the thumbnail display area 802 at once, they can be sequentiallydisplayed by dragging in the scroll bar. Numeral 804 denotes a pointer.The user moves the pointer 804 by operation at the user operation unit300, to designate a desired thumbnail in the thumbnail display area 802.When the thumbnail is selected by the pointer 804, the display screenimage changes to a moving-image reproduction screen image 805 as shownin FIG. 8B, and moving image reproduction is started.

Hereinbelow, processing to realize the utilization of thumbnail in abrowser will be described. FIGS. 9A and 9B are flowcharts showingprocessing upon browser operation in the moving image processingapparatus of the present embodiment. Note that the processing shown inFIGS. 9A and 9B is executed when browser processing is started in themoving image processing apparatus and a moving image file to be browsedis designated. Note that upon designation of moving image file to bebrowsed, the thumbnails representing the respective moving image files(thumbnails stored in the headers of the respective moving image files)are displayed in the thumbnail display area 802 in FIG. 8A. Then, adesired moving image file is selected by designating a desired thumbnailby the pointer 804. When the desired moving image file is selected, theprocessing shown in FIGS. 9A and 9B is started.

First, at step S201, the browse unit 304 reads the header 401 of themoving image file 400 selected by the above operation among the movingimage files stored in the moving image file storage unit 302. Then thebrowse unit obtains various basic management information such as thefile size from the moving image file management information 404, andstores the information into the memory (RAM 203). Next, at step S202,the unit reads the index area 406 in the footer 403 and stores it in thememory (RAM 203). Next, at step S203, the unit performs sorting on theindexes included in the index area 406 held at step S202 by startposition (Start Point in FIG. 5) in ascending order, for display ofthumbnails to be described later in time sequential order.

Next, at step S204, a thumbnail display start number N (indicating thatthe N-th thumbnail is at the left end of the top row) is obtained fromthe current bar position of the scroll bar 803. For example, if thescroll bar 803 is at an upper end (as shown in FIG. 8), display startnumber N=1 holds. Accordingly, display is started from the firstthumbnail.

Next, by performing the following processing at steps S205 to S212sequentially from the head index, thumbnails for browsing are displayedin the thumbnail display area 802 from the N-th thumbnail.

First, at step S205, it is checked whether or not the thumbnail isappropriate to browsing based on the selection base upon registration ofthumbnail. Even in the same usage of browsing, the base of thumbnail asan appropriate index in browsing operation may differ in accordance withdevice. In the present embodiment, the browse unit 304 determines thatthe head frame in the section is appropriate as an index thumbnail. Tocheck whether or not it is a head frame in the section, it is checkedwhether Thumbnail Type ID=1 holds. Note that upon start of processing,checking is started from the head after the sorting.

If it is determined at step S205 that it is a thumbnail for browser, theprocess proceeds to step S206. At step S206, the number C (the order inthe thumbnails for browser) is counted. At step S207, it is checkedwhether or not the counted number C is less than the display startnumber N (C<N). This processing is made for skipping thumbnails beforethe number N without display.

If the number C is equal to or greater than the display start number N,the thumbnail image is read for thumbnail display at step S208. Next, atstep S209, the thumbnail image is displayed in the thumbnail displayarea 802, and the process proceeds to step S210. On the other hand, ifit is determined at step S205 that the thumbnail is not for browsing, orif it is determined at step S207 that the number C is less than thenumber N, the process proceeds to step S210.

At step S210, the processing subject is moved to the next index toprocess the next thumbnail. In a case where the immediately previousprocessing has been performed on the last index and there is no nextindex, the process proceeds to step S213. If the next index exists, theprocess proceeds to step S212, at which it is checked whether or not thenumber has exceeded the number of displayable thumbnails in thethumbnail display area 802. For example, in the example in FIG. 8A, thenumber of displayable thumbnails is 9. Accordingly, if C≧N+9 holds, itis determined that displayable thumbnails have been displayed.

If the number is not over the number of displayable thumbnails, theprocess returns to step S205. That is, the processing at steps S205 toS210 is performed to sequentially check the indexes until there is noindex to be processed or the number exceeds the number of displayablethumbnails.

If there is no index to be processed or the number exceeds the number ofdisplayable thumbnails in the thumbnail display area 802, the processproceeds to step S213. At step S213, it is checked whether or not thescroll bar has been operated by the user. If it has been operated, theprocessing at steps S204 to S212 is repeated, thereby a thumbnailcorresponding to the position of the operated scroll bar is displayed.If the scroll bar has not been operated, the process proceeds to stepS214.

Next, at step S214, it is determined whether or not one of thethumbnails displayed in the thumbnail display area 802 has been selectedby the user. If a thumbnail has been selected, the process proceeds tostep S215, at which the Start Point is read from the index correspondingto the selected thumbnail. Next, at step S216, moving image reproductionis started from the Start Point, and the moving image is displayed inthe moving image reproduction screen image 805. Thereafter, the processproceeds to step S217. On the other hand, if it is determined at stepS214 that a thumbnail has not been selected, the process proceeds tostep S217.

At step S217, it is determined whether or not termination of browsing bythe browse unit 304 has been instructed. If it is determined thattermination has not been instructed, the process returns to step S213.If it is determined that termination has been instructed, the processends.

Note that in the above example, when a thumbnail is selected, the StartPoint is obtained from the corresponding index and reproduction isstarted from the position and continued by the end of the moving imagedata. However, the reproduction is not limited to this arrangement. Itmay be arranged such that the Start Point and the End Point are obtainedfrom the index corresponding to the selected thumbnail and only thesection is reproduced. Note that a stop or pause button may be providedsuch that moving image reproduction can be stopped in an arbitraryposition.

In the above description, browsing is made in one moving image file forthe sake of simplicity of explanation, however, even in thumbnailselection in plural moving image files, the selection can be made byrepeating the processing to display a thumbnail for the respectivemoving image files.

By the above processing, the browse unit 304 reads only thumbnail(s) forbrowsing from the thumbnail area 407 and utilizes it. That is, theapparatus or processing utilizing thumbnails can select a thumbnailappropriate to the processing by the information upon generation ofthumbnail (Thumbnail Type ID), thereby can utilize the thumbnail.

[Utilization by Moving Image Editing]

Next, utilization of thumbnail in moving image editing will bedescribed. FIG. 10 is an example of the user interface upon moving imageediting by an editing unit 305. Numeral 1000 denotes an editingoperation window in which an editing operation is performed. Numeral1001 denotes a reproduction window in which a selected moving image fileor the result of editing is reproduced for checking. Numeral 1002denotes a moving image file window for display of a list of thumbnails(significant thumbnails) in the headers of the respective moving imagefiles. Numeral 1003 denotes operation buttons for operation of movingimage reproduced in the reproduction window 1001, for “reproduction,temporary stoppage, stoppage, reverse and fast-forward” operations fromthe left.

Numeral 1004 denotes a scene designation window in which selected movingimage files are arrayed for designation of section in the respectivemoving images. Note that thumbnail images are overlay-displayed in therespective moving image file areas for the user's recognition. In thefigure, files represented by thumbnails of “sun” and “baseball player”are arrayed. Numeral 1005 denotes a scene designation bar. The startpoint and the end point of selected clip (hatched portion) can bedesignated by operating the scene designation bar.

Numeral 1006 denotes a pointer. The user moves the pointer 1006 byoperating the user operation unit 300, to designate selection of variousbuttons and designation of operation. In an operation procedure, amoving image files are selected from the moving image file window 1002and dragged and dropped into the scene designation window 1004 in theorder of reproduction. Then, the start point and the end point arecorrected by using the scene designation bar 1005. At this time, theuser performs moving image reproduction in the reproduction window 1001by operating the operation unit 1003 in accordance with necessitythereby checks the image.

FIG. 11 is a flowchart showing a thumbnail display operation upon themoving image editing by the editing unit 305. Note that the processingin FIG. 11 is started when moving image editing processing is executedin the moving image processing apparatus.

First, at step S300, the editing unit 305 reads the headers 401 of allthe moving image files from the moving image file storage unit 302.Next, at step S301, the editing unit reads the thumbnails 405 in theheader 401 of the respective files. Then at step S302, the editing unitdisplays the thumbnails in the headers (i.e. significant thumbnails) asrepresentative images of the respective moving image files in the movingimage file window 1002.

Next, at step S303, it is determined whether or not a moving image filehas been selected. In the present embodiment, it is determined whetheror not the moving image file(s) has been dragged and dropped from themoving image file window 1002 to the scene designation window 1004 byoperation of the pointer 1006. If moving file(s) has been selected, theprocess proceeds to step S304, at which the index area 406 is read fromthe footer in the moving image file selected by the user. On the otherhand, if it is determined at step S303 that moving image file selectionhas not been made, step S304 is skipped.

Next, at step S305, it is checked whether or not the user has designateda range of scene regarding the moving image file by the scenedesignation bar 1005 in the scene designation window 1004. If the rangedesignation has been made, thumbnail(s) for editing existing within therange designated by the user is extracted and displayed at steps S306 toS311.

First, at step S306, the Start Point and the End Point in the index(upon start of processing, the head index) are referred to, and it isdetermined whether or not a section indicated by the Start Point and theEnd Point is within the range designated by the user. If it is withinthe range, the process proceeds to step S307, at which it is determinedwhether or not the thumbnail corresponding to the index is a thumbnailappropriate to editing based on the selection base of the thumbnail.Even in the same usage of editing, the base of thumbnail appropriateindex in editing may differ in accordance with device. The editing unit305 of the present embodiment determines a middle frame in the sectionas a thumbnail appropriate to editing. To check whether or not it is amiddle frame in the section, it is checked whether Thumbnail Type ID=2holds. If it is a thumbnail appropriate to editing, the process proceedsto step S310.

On the other hand, if it is determined at step S306 that the sectioncorresponding to the index is not within the range designated by theuser, or if it is determined at step S307 that the thumbnail is not forediting, the process proceeds to step S308 to process the next index. Atstep S309, it is determined whether or not the next index exists. If thenext index exists, the processing at steps S306 and S307 is repeated.

If the condition at step S307 is satisfied, the process proceeds to stepS310, at which a thumbnail image corresponding to the index is read.Then at step S311, the thumbnail image is displayed. For example, asshown in FIG. 10, in the scene designation window 1004, the thumbnailimage is overlay-displayed on a bar indicating the scene. By thisdisplay, the user can easily recognize the scene in the designatedrange. Thereafter, the process proceeds to step S308, to repeat theabove-described processing on the next index.

By the above processing, thumbnails for editing in all the sectionswithin the range designated by the user are overlay-displayed on thebars indicating the scenes. Note that the display range of thumbnail isan area designated by 2 scene designation bars 1005. If the size of areabetween the designation bars is insufficient, a part of the thumbnailmay be overlaid or a scroll bar may be displayed for scrollable display.

If it is determined at step S305 that a scene range has not beendesignated by the user, or if it is determined at step S309 that thereis no index to be processed, the process proceeds to step S312 At stepS312, it is determined whether or not termination of the moving imageediting operation has been instructed. If the termination of the editingoperation has not been instructed (the moving image editing isperformed), the process returns to step S303 to repeat the aboveprocessing. On the other hand, if it is determined that the terminationof the editing operation has been instructed, the process ends.

By the above processing, upon file selection, the editing unit 305 canuse a significant thumbnail stored in the header, and utilize onlythumbnails for moving image editing corresponding to the designatedrange from the thumbnail area 407.

Note that in the above embodiment, the browsing and moving image editinghave been described as utilization of thumbnail images, however, theusage is not limited to these 2 operations, but thumbnail images can beused for various purposes such as printing of frame image correspondingto a selected thumbnail.

For example, if a frame image corresponding to a selected thumbnail isprinted, thumbnails appropriate to printing are extracted fromthumbnails corresponding to designated image data (thumbnails recordedin the footer) and displayed as printing frame candidates, and a frameas an original of the selected thumbnail is printed. Even in the sameusage of printing, the base of thumbnail appropriate to printing maydiffer in accordance with device. For example, if a gentle motion frameand a high-image-quality frame in the section are handled as framesappropriate to printing, it is checked whether Thumbnail Type ID=5 orThumbnail Type ID=8 holds. When a frame is designated, an image of theframe specified by the Thumbnail Key Relative point shown in FIG. 6 ofthe designated thumbnail is obtained and is printed. Accordingly,another function such as a printing unit may be added to the movingimage processing apparatus in FIG. 3.

Note that in the above embodiment, the browsing unit and the editingunit are provided as functions for utilizing thumbnails, however, onlyone of these functions including the above printing unit, or otherplural functions may be used.

Note that in the above embodiment, the moving image processing apparatusshown in FIG. 3 is a single apparatus including the moving image inputunit 301, the thumbnail generation unit 303, the browse unit 304, andthe editing unit 305. However, the moving image processing apparatus maybe constructed as plural apparatuses of respective functions, includingthe other unit such as the above-described printing unit, or apparatuseshaving combinations of several functions. In such case, a moving imagefile 400 (FIG. 4) including thumbnails may be transmitted among therespective apparatuses via a storage medium or communication means suchas IEEE 1394.

Note that in the above embodiment, since there is no index for theheader thumbnails, the header thumbnails do not have the Start Point,the End Point and the Thumbnail Type ID, however, these information maybe held as management information.

Note that in the above embodiment, the thumbnail recorded in the header(thumbnail representing the file) is not recorded in the footer, but thethumbnail may be recorded in the header and the footer.

Note that in a case where thumbnails generated from plural selectionbases are the same (for example, a middle frame and a dynamic motionframe in the section are the same), the thumbnails may be registeredwith different Thumbnail Type IDs, or one Thumbnail Type ID (e.g.Thumbnail Type ID=10: middle and dynamic motion frame in section) may beprepared. In the latter case, as it is not necessary to register thethumbnails which are the same, the efficiency of data use can beimproved. Note that it may be arranged such that plural selection bases(Thumbnail Type IDs) can be registered for one thumbnail.

Note that in the above embodiment, the registration of thumbnail isdetermined by the user's instruction, however, it may be automated. Forexample, to automatically extract a dynamic motion frame in a section, acurrent frame and a previous frame are divided into plural blocks invertical and horizontal directions as shown in FIG. 12. In each block,RGB mean values are calculated, and the sum of squares of thedifferences in respective RGB channels is obtained between the currentand previous frames as an inter-frame similarity distance. An example ofthe expression is as follows.

$\begin{matrix}{\sum\limits_{i = 1}^{k}\left\{ {\left( {{P\; 1_{iR}} - {P\; 2_{iR}}} \right)^{2} + \left( {{P\; 1_{iG}} - {P\; 2_{iG}}} \right)^{2} + \left( {{P\; 1_{iB}} - {P\; 2_{iB}}} \right)^{2}} \right\}} & \left\lbrack {{Expression}\mspace{20mu} 1} \right\rbrack\end{matrix}$i: block being processedK: the number of divided blocksP1 _(iR): R channel mean value in i-th block of previous frameP1 _(iG): G channel mean value in i-th block of previous frameP1 _(iB): B channel mean value in i-th block of previous frameP2 _(iR): R channel mean value in i-th block of current frameP2 _(iG): G channel mean value in i-th block of current frameP2 _(iB): B channel mean value in i-th block of current frame

The frames are similar as the inter-frame similarity distance is small,while the frames are not similar as the distance is large. As a motionis dynamic, similarity between the frames reduces, a frame having aninter-frame similarity distance equal to or greater than a thresholdvalue may be determined as a dynamic motion frame.

As described above, according to the present embodiment, as pluralthumbnails corresponding to plural purposes and selection bases can bestored as one file, the thumbnails can be easily managed. Further, uponuse of thumbnail, only a thumbnail corresponding to the purpose and adevice feature to utilize the thumbnail can be used by referring to theselection base (Thumbnail Type ID) upon generation of thumbnail added tothe thumbnail.

Further, the object of the present invention can be also achieved byproviding a storage medium holding software program code for realizingthe functions of the above-described embodiment to a system or anapparatus, and reading and executing the program code stored in thestorage medium by a computer (e.g., CPU, MPU) of the system orapparatus.

In this case, the program code read from the storage medium itselfrealizes the functions according to the embodiment, and the storagemedium constitutes the invention.

Further, the storage medium, such as a floppy disk, a hard disk, anoptical disk, a magneto-optical disk, a CD-ROM, a CD-R, a DVD, amagnetic tape, a non-volatile type memory card, and a ROM can be usedfor providing the program code.

Furthermore, besides aforesaid functions according to the aboveembodiment are realized by executing the program code by the computer,the present invention includes a case where an OS (operating system) orthe like working on the computer performs a part or entire actualprocessing in accordance with designations of the program code andrealizes the functions according to the above embodiment.

Furthermore, the present invention also includes a case where, after theprogram code read from the storage medium is written in a functionexpansion board which is inserted into the computer or in a memoryprovided in a function expansion unit which is connected to thecomputer, a CPU or the like contained in the function expansion board orunit performs a part or entire actual processing in accordance withdesignations of the program code and realizes the functions of the aboveembodiment.

As described above, according to the present invention, operability ofvarious moving image processing using a representative image is improvedby adding information regarding generation of representative image asits attribute to the representative image, and management ofrepresentative images can be simplified.

As many apparently widely different embodiments of the present inventioncan be made without departing from the spirit and scope thereof, it isto be understood that the invention is not limited to the specificembodiments thereof except as defined in the appended claims.

1. An apparatus for managing one or plural representative images incorrespondence with a moving image, comprising: a unit configured tostore a representative image, generated based on a frame image selectedfrom a section in the moving image, in correspondence with the sectionin the moving image; a unit configured to store section informationindicating the section in the moving image and attribute informationindicating a selection base of the frame image, in correspondence withthe representative image; and a selection unit configured to acquire arepresentative image corresponding to a processing purpose by selectinga representative image from representative images stored incorrespondence with a desired moving image, whose attribute informationindicates a selection base corresponding to the processing purpose. 2.The apparatus according to claim 1, wherein the moving image, therepresentative image, the section information, and the attributeinformation are stored as one file.
 3. The apparatus according to claim1, further comprising a unit configured to store position informationindicating a position of the frame image in correspondence with therepresentative image.
 4. The apparatus according to claim 1, wherein thesection information and the attribute information are recorded in aparticular continuous portion by representative image.
 5. The apparatusaccording to claim 4, wherein the particular portion is a continuousarea on the storage medium.
 6. The apparatus according to claim 1,further comprising an application unit configured to, if arepresentative image selected by said selection unit is designated,apply processing corresponding to the processing purpose for a movingimage in a section indicated by the section information corresponding tothe designated representative image.
 7. A method for managing one orplural representative images in correspondence with a moving image,comprising: a step of storing a representative image, generated based ona frame image selected from a section in the moving image, incorrespondence with the section in the moving image; a step of storingsection information indicating the section in the moving image andattribute information indicating a selection base of the frame image, incorrespondence with the representative image; and a selection step ofacquiring a representative image corresponding to a processing purposeby selecting a representative image from representative images stored incorrespondence with a desired moving image, whose attribute informationindicates a selection base corresponding to the processing purpose. 8.The method according to claim 7, wherein the moving image, therepresentative image, the section information, and said attributeinformation are stored as one file.
 9. The method according to claim 7,further comprising a step of storing position information indicating aposition of the frame image in correspondence with the representativeimage.
 10. The method according to claim 7, wherein the sectioninformation and the attribute information are recorded in a particularcontinuous portion by representative image.
 11. The method according toclaim 10, wherein the particular portion is a continuous area on thestorage medium.
 12. The method according to claim 7, further comprisingan application step of, if a representative image selected at saidselection step is designated, applying processing corresponding to theprocessing purpose for a moving image in a section indicated by thesection information corresponding to the designated representativeimage.
 13. A program stored on a computer-readable medium for making acomputer perform the method according to claim
 7. 14. Acomputer-readable medium for storing a program for making a computerperform the method according to claim 7.